Tool tracking

ABSTRACT

A system for determining the location of a toolpiece, the system comprising: a tool carrying a toolpiece and comprising an imaging device for capturing images of the environment around the tool, the tool being actuable to manipulate the toolpiece to work on a workpiece; and an image processor remote from the tool and communicatively coupled to the imaging device for receiving images therefrom and having access to reference data representing an expected environment, the image processor being configured to compare an image captured by the imaging device with the reference data to identify a match therebetween and to determine in dependence on characteristics of that match the location of the toolpiece; wherein the system is configured to cause the tool to transmit images captured by the imaging device to the image processor in response to actuation of the tool.

This invention relates to estimating the location of an object. Theobject could, for example, be a tool.

The best known approach for automatically determining the location ofobjects is the use of satellite positioning systems such as GPS. Thesesystems can provide an accuracy of a few metres in outdoor environments.However, there are situations in which satellite positioning systems areunable to provide enough accuracy. One such situation is when the objectwhose location is to be estimated is indoors, and so reception ofsignals from the satellites is impaired. Another is when the position ofthe object needs to be known with an accuracy greater than can readilybe achieved through satellite positioning. Some specific examples ofsituations where satellite positioning is inadequate are: (i) trackingthe movements of individuals who are interacting with a video gameconsole, and (ii) tracking the movement of tools in a factory to monitorproductivity or to verify that the correct processes are being performedon an item that is being manufactured. Specialised systems have beendeveloped for these applications.

In one example of such a system a set of cameras are placed at fixedlocations around the environment where an object is to be located. Thecameras capture a video stream or a series of still images. Imageprocessing software analyses the cameras' output to recognise the objectthat is to be monitored. By correlating the position of the object asseen by the cameras with the location of the cameras the position of theobject can be estimated in three dimensions. The image processingsoftware might be capable of recognising the object itself in thevideo/image feed. An example of a system of this type is the MicrosoftKinect gaming system. Alternatively, the object could carry a visuallydistinctive fiducial marker that is identifiable by the software. Anexample of this type of system is the Sony PlayStation Move MotionController.

In another example, the object to be located carries an acoustictransmitter. A set of acoustic receivers are placed at fixed locationsaround the environment where the object is to be located. The systemmeasures the time taken for signals transmitted from the object to bereceived at a number of the receivers. Then, by correlating the time offlight from the transmitter to each of those receivers with the locationof the receivers the position of the object can be estimated in threedimensions. An example of this type of system is the Sarissa LocalPositioning System. In a similar way, other systems employ a radiotransmitter carried by the object to be measured, which cooperates withmultiple radio receivers located around the environment. The radioreceivers may detect any of a range of parameters (for exampletime-of-arrival, time-difference-of-arrival, angle-of-arrival or signalstrength) of radio signals sent by the transmitter. Using theseparameters the location of the transmitter relative to the receivers canagain be estimated.

Systems of the types described above are generally able to locateobjects to an accuracy of a few tens of centimetres. This level ofaccuracy is sufficient for many applications. For example, a gamesconsole can operate successfully knowing the position of a user to thatdegree of accuracy. Similarly, on an automotive production line thatdegree of accuracy is sufficient to establish which vehicle on theproduction line a particular tool is being used on. That informationcould be used to monitor the productivity of a person using the tool. Itcould also be used to control the operation of the tool. For example,the tool may be an automated wrench being used to assemble engine mountsin a vehicle. The engine mounts may need to be tightened to differenttorques depending on the type of engine being installed. By monitoringthe location of the tool, and thereby determining which vehicle on theproduction line the tool is being used on, it is possible toautomatically configure the torque setting on the tool to theappropriate value.

However, there are a large number of other applications which demand aneven higher level of accuracy, perhaps down to one centimetre in threedimensions. For example, assembly of a helicopter gearbox requires alarge number of closely-spaced bolts to be tightened in a precisesequence. It would be useful to be able to automatically verify that theassembly has been completed correctly by monitoring the location of thetool being used to tighten the bolts. To do this the monitoring systemwould need to be able to determine, preferably in real time, theposition of the tool head accurately enough to distinguish betweenoperations on neighbouring bolts which may be only a centimetre or twoapart. Similar situations arise in a car production plant in installingcylinder heads and assembling transmission housings.

Bolt-level positional tracking is difficult to achieve with anyreliability using the systems described above, especially in aproduction environment. In addition to requiring a high level ofpositional accuracy a further problem is that during an assembly processbolts are often obscured inside other structures, making it difficult toestablish the location of the tool head when it engages the bolt.

According to one aspect of the present invention there is provided asystem for determining the location of a toolpiece, the systemcomprising: a tool carrying a toolpiece and comprising an imaging devicefor capturing images of the environment around the tool, the tool beingactuable to manipulate the toolpiece to work on a workpiece; and animage processor remote from the tool and communicatively coupled to theimaging device for receiving images therefrom and having access toreference data representing an expected environment, the image processorbeing configured to compare an image captured by the imaging device withthe reference data to identify a match therebetween and to determine independence on characteristics of that match the location of thetoolpiece; wherein the system is configured to cause the tool totransmit images captured by the imaging device to the image processor inresponse to actuation of the tool.

Potential aspects of the present invention are set out in theaccompanying claims.

The image processor may be provided in the form of a singlemicroprocessor or its functions may be distributed between multiplemicroprocessors at the same or different locations.

The present invention will now be described by way of example withreference to the accompanying drawings. In the drawings:

FIG. 1 shows an example of a positioning system for a tool.

FIG. 2 shows a tool employed on a production line.

In the systems to be described below, a tool is equipped with one ormore sensors that are capable of sensing the environment immediatelyaround the tool. The output of those sensors is compared with pre-storedexpected characteristics of the environment. Based on that comparisonthe location of the head of the tool can be determined with sufficientaccuracy to be able to know which bolt in an assembly the tool isengaged with.

FIG. 1 shows a specific example of this system. A powered wrench 1 has abody 2 intended to be held by a user and a head section 4 for performingdriving operations. The head section 4 includes a motor housing 3,comprising a drive motor 13, a drive shaft 5 which protrudes from themotor housing and a toolpiece 6. The motor 13 is capable of driving theshaft 5 and hence the toolpiece 6 to rotate relative to the motorhousing 3. The motor 13 could be at least partially located in the body2. In this example the toolpiece is a hex socket, but it could take anyother suitable form. The hex socket 6 is sized to engage bolts 7, 8, 9which form part of a workpiece 10. The bolts could all be intended tohold a common pair of components together.

The tool comprises a battery 11 which powers its operations.Alternatively, the tool could be powered by an electrical wire connectedto a mains electricity supply. Or the tool could be a pneumaticallydriven tool, having an electrical supply (e.g. from a local battery) topower only its computation operations.

The tool has a motor controller 12 which controls the operation of themotor 13 under the command of an operator switch 14. The motor isconfigured to apply a torque that is limited by the present setting of atorque limiter 15. The torque limiter could be an electrical controllerthat limits the performance of the motor 13 (e.g. via the motorcontroller 12), or it could be a mechanical device such as a slip clutchlocated in the output shaft 5. In the case of a pneumatic tool, thetorque limiter could similarly operate to restrict the developed drivetorque or to relieve torque in the output shaft.

The tool comprises a camera 16. In this example the camera is attachedto the body 2 of the tool, but it could be attached elsewhere on thetool. When the tool is in operation the camera 16 captures a videostream depicting the environment around the tool. The video stream ispassed to a local control unit 17. The local control unit transmits thevideo stream wirelessly via an antenna 18 and a remote antenna 19 to aremote processing unit 20. The remote processing unit 20 can process thevideo stream in a manner that will be described below in order toestimate the location of the tool, specifically the head 4 of the tooland most specifically the toolpiece 6.

Before the tool 1 is to be used to work on the workpiece 10, the systemis configured by storing in a database 21 a set of images of the workingenvironment on and around a reference workpiece. These images can becaptured using a conventional hand-held camera or using a camera mountedon a tool like tool 1. The reference workpiece can simply be aproduction workpiece chosen at random to serve as an exemplar. Theimages may include images of the workpiece in different configurations,for example with different numbers or sets of sub-parts (e.g. bolts 7-9)installed and in different lighting conditions. The images stored indatabase 21 may include images of the working environment on and aroundmultiple different workpieces (for example different types of cylinderheads, different seats or colours of seats), and for workpieces indifferent surroundings (for example seats installed in different carbodies).

When the processor 20 receives a video feed from the tool 1 it attemptsto match frames of that feed to the images stored in the database 21. Itdoes this by comparing the images, under a range of geometrictransformations, with the stored images. The transformations can includeany one or more of:

-   scaling transformations, to enable a match to a similar part of the    imaged environment captured at a different viewing range;-   trapezoid transformations, to enable a match to a similar part of    the imaged environment captured at a different viewing angle;-   translational transformations, to enable matches between different    parts of the captured and stored images;-   rotational transformations, to enable a match to a similar part of    the imaged environment captured at a different orientation; and-   brightness, contrast and colour transformations, to enable a match    to a similar part of the imaged environment captured under different    lighting conditions or to enable a match to a similar shaped part    but of a different colour or material.

The processor 20 applies a range of these transformations, alone or incombination, to received frames of the captured image stream, and thenscans the stored images to identify a match between part or all of thecaptured image and part or all of one of the stored images. To simplifythe process of matching the images, the process of determining a matchmay be performed on monochrome versions of the images, or may beperformed by comparing the location of high contrast edges or otherdistinct features in the captured image and the stored image. Otherimage comparison techniques can be used. A match may be deemed to existwhen there is greater than a pre-determined level of similarity betweenpart or all of a captured image and part or all of a stored image. Thatsimilarity could be defined by respect to numbers or proportions ofpixels, blocks of pixels, or image features such as edges. The imagesmay include many fine-scale features that in isolation have a similarappearance. Therefore, the processor 20 is preferably configured toaccept a match only when a sufficient area of image is matched, or onlywhen image portions exceeding a pre-set level of complexity match eachother.

The database 21 holds metadata for each of the images it stores. Thatmetadata includes information identifying the location of the image: forexample the three-dimensional coordinates of one or more referencepoints in the image. The coordinates can conveniently be by reference toa defined location on the relevant workpiece. The metadata may alsoinclude an indication of the workpiece to which the image relates: forexample to a particular car body or engine component.

Once the processor 20 has identified a match between the captured imagefrom the tool and a stored image, it can then determine the location ofthe toolpiece 6. To do that it determines an offset from one or more ofthe reference points of the image in dependence on (i) thetransformations applied to the image in order to achieve the match and(ii) the characteristics of the camera 16 and the tool 1, for examplethe viewing angle of the camera's lens and the spatial relationshipbetween the camera 16 and the toolpiece 6. The characteristics of thecamera and the tool can be pre-stored in memory 22 in the processor 20.The memory 22 may store that data for multiple types of tool, and may beinformed of the type or identity of the particular tool 1 by atransmission from the local controller 17. Instead of determining anoffset from a reference point in the reference image, the metadata couldstore the location from which the reference image was captured and theprocessor 20 could determine the location of the toolpiece 6 as anoffset from the image capture reference point in an analogous way tothat used to determine the location based on an offset from a referencepoint in the image itself.

The reference images preferably depict exclusively or predominantlyelements of the expected environment of the workpiece that are staticwith the workpiece during expected use of the tool. In that way thelocation of the toolpiece can readily be determined as a locationrelative to the workpiece. That simplifies the task of establishingwhich part of the workpiece the toolpiece is working on.

Since the camera 16 can capture images of the environment around thetool, the location of the toolpiece 6 can be determined relativelyaccurately: for example to within a few centimetres. The camera isparticularly effective when the tool is being operated in closeproximity to other parts, as for example when it is being used in aconfined space. Then the camera can be relatively close to the visiblefeatures around the tool, meaning that the image comparison operationcan yield a particularly accurate tool position. When the toolpiece canbe located to within a few centimetres it is possible to determine whichpart of a workpiece it is operating on: for example which of a number ofadjacent bolts, nuts, screws or studs it is driving.

Once the location of the toolpiece 6 is known, that location can be usedfor a range of purposes.

In one example it can be used to verify that assembly operations arebeing performed correctly (e.g. that bolts are tightened in a correctsequence). To do this, the locations of the tool piece over time can bestored by the processor 20 and compared with a reference workingpattern. An alert can be raised if the sequence of locations of the toolpiece deviates materially from the reference working pattern. The toolmay also transmit to the processor 20 an indication of when the motor 13is being driven and/or the applied drive torque. That information mayalso be compared with the reference working pattern, allowing theprocessor 20 to determine not just that the toolpiece has visited therequired locations but that bolts have been tightened in thoselocations, that the bolts have been tightened in the correct order andthat each one has been tightened to the correct torque. In this way, itis possible to verify that the bolts of a component such as a cylinderhead have been tightened in the correct order and/or to the correcttorque, or that all the bolts of a component such as a sump pan havebeen tightened individually (irrespective of order).

In another example, the location of the tool can be used to control theoperation of the tool. It may be that different parts of a singleworkpiece need to be worked on with different settings of the tool. Forexample, different bolts may require tightening to different torques orwith different drive speeds, or some bolts may have reverse threads andneed to be tightened by anti-clockwise operation of the motor 13. Theprocessor 20 can store in memory 22 a map of tool settings for differentlocations on the workpiece. Then when the toolpiece is located at aparticular location it can transmit to the local controller 17 a settingto be applied to the tool. In the case of a torque setting thecontroller 17 can then set the torque limiter 15 accordingly. Othersettings could be applied in different ways, for example by the localcontroller 17 signalling the motor controller 12 to function in adesired way. In this way the tool can automatically be adapted toworking requirements for different points on a single workpiece. Thetool settings may be sent from the processor 20 automatically when itdetermines that the toolpiece 6 is at the location of a part on theworkpiece for which it stores setting information. Alternatively, thelocal controller may be configured to request settings from theprocessor 20 automatically in response to the user actuating theoperating switch 14 or in response to the tool detecting that it isengaged with a part of the workpiece (e.g. by means of a proximitysensor in the toolpiece).

A workpiece such as a car engine may have multiple bolts (separated bysmall distances, perhaps only 1 cm or so) which need tightening at anyparticular stage of production, and those bolts may have differenttorque requirements. By knowing to a bolt-level precision which bolt thetool is attached to at any one time before the tightening operation isperformed the tool can be set to the appropriate torque. This improvesmanufacturing efficiency and reliability.

In summary, one or more cameras 16 are attached to the tool 1 at knownpoints. Images captured by the camera(s) are sent to a computation unit20, which compares the captured images with the stored images in thedatabase 21. The comparison of the captured images and the stored imagesmay be by image correlation or other comparison techniques. Thecomparison can take account of the potential scale, rotation,translation and perspective transformations due to the relative positionof the camera(s) 16 and the environment around it/them. When a match(for a given scale, rotation, translation and perspective) between thecaptured and stored images is determined by the computation unit 20, theparameters of those transformations can be used to calculate, bystandard geometric methods, the 3D location and orientation of thecamera(s) 16 relative to its/their environment. In turn, using the knownposition and orientation of the camera(s) 16 relative to the head of thetool 4, 6, and the measured position and orientation of the camera(s) 16relative to its/their environment, the position of the head of the tool4, 6 relative to the workpiece 7, 8, 9, 10 can be found. When the tool 1is activated to begin tightening a bolt 7, 8, 9, the location of thehead of the tool 4, 6 can be compared by the computation unit 20 withthe locations of the bolts stored in memory 22, and the desired torquesetting for the closest match can be retrieved from memory 22 andprogrammed into the tool 1. After the tightening operation is complete,the achieved torque for the bolt 7, 8, 9 whose location most closelymatched the position of the head of the tool 4, 6 can also be recordedby the computation unit 20.

The camera 16 could be a time-of-flight (TOF) camera. A TOF camera isable to sense the depth (i.e. the distance away from the focal plane ofthe camera) of points in the images it captures. In addition to, orinstead of, storing visual images of the exemplar environment/workpiecethe database 21 can store one or more three-dimensional maps of theenvironment and/or the workpiece suitable for matching to thethree-dimensional data captured by the TOF camera. Then the processor 20can match the captured data to the stored reference data in an analogousway to that previously described, and thereby derive the location of thetoolpiece 6. The processor could perform matching using both visibleimages and three-dimensional data.

The ability of the processor 20 to match the images from the camera 16to the images stored in the database 21 will depend on the degree ofdifferentiation in the workpiece and its environment. If the workpieceand the environment are particularly uniform, or if similar parts of theworkpiece or the environment have similar appearances (e.g. as with aset of bolts surrounding a uniform round shaft), the workpiece or theenvironment around it could be adapted to permit the tool to be locatedmore accurately. For example, the workpiece could carry some visiblefeature or surface relief formation to differentiate one part fromanother. Such a feature or formation could be moulded, cast or tooledinto the workpiece, or could be applied to the workpiece by printing oretching or applying an adhesive marker. Furthermore, the working patterncould be chosen to assist in locating the tool. The working patterncould be to first install one or more bolts at places where the locationof the tool is initially differentiable, and then to install other boltsat places where the location of the tool is differentiable by referenceto already-installed bolts.

The determination of the location of the tool and/or the toolpiece maybe assisted by other techniques, as will now be described.

The tool 1 could comprise an inertial motion detector 23. The inertialmotion detector could use gyroscopes and/or accelerometers to detectmotion of the tool. The output of the motion detector 23 is passed tothe controller 17. The controller 17 can transmit the detected motion tothe processor 20. The processor 20 can use the motion data in a numberof ways. First, it can use the motion data to help analyse the imagescaptured by the camera with a view to matching them to the stored imagedata. The set of transformations to be applied to the captured image topermit it to be matched to a stored image could be selected, or certainselected transformations could be prioritised, in dependence on themotion data. Similarly, which of the stored images the captured image isto be compared to may be selected in dependence on the motion data. Thismay reduce the time needed to match the stored image to a capturedimage. Second, it can use the motion data to estimate the location ofthe tool for workflow purposes if the position of the tool cannot bedetermined for the time being by means of the captured images. Althoughthe inertially-derived motion data may be expected to degrade inaccuracy over time, during a short interval between the toolpiece beinglocated by other means an inertially-derived position may be adequatefor selecting settings of the tool or recording the activities of thetool.

In another example, the tool may be being used in a production facilitywhere the state or environment of a workpiece changes as it movesthrough the facility along the line. For example, the workpiece may be avehicle body on a production line. As the body progresses along theproduction line increasing numbers of ancillary parts such as theengine, running gear, interior trim and doors are attached to the body.These change the appearance of the body and its environment. Thedatabase 21 may store many images of a reference body in differentstates along the production line. When the processor 20 receives animage from the tool 1 it must match the captured image against one ofthose images. That may require a considerable amount of processing ifthere are many stored images. To simplify that processing, the processor20 may select or prioritise a subset of the stored images for comparisonwith the captured image in dependence on a coarse position of the toolas determined by another mechanism. This is illustrated in FIG. 2.

FIG. 2 shows a car production line 37. Car bodies 30 move along theproduction line, and additional components such as wheels 31, engines 32and doors 33 are added to the bodies as they progress along the line.Tool 1 is being used to assemble the cars. Tool 1 comprises a radiotransmitter 24 (see FIG. 1) which cooperates with radio receivers 34,35, 36 located at known locations around the production line. Thesignals received by the radio receivers can be processed to estimate thelocation of the tool, for example by measuring the time-of-arrival andangle-of-arrival at the receivers of signals from the transmitter 24.Typically that might yield the location of the tool to within a few tensof centimetres. Other coarse positioning techniques could be used toachieve a similar result. Once the coarse location of the tool is known,the processor 20 can estimate the state of the vehicle body at thatlocation. Metadata stored with the images in database 21 may indicatethe state of production and/or the coarse location on the productionline to which they relate. The processor 20 can then select orprioritise for comparison with the captured images the stored imagesthat relate to the coarse location on the production line where the toolcurrently is. In the example of FIG. 2, that means the processor willpreferentially attempt to match an image captured by the tool withimages that pertain to the body 30 with wheels 31 and engine 32installed but without doors 33.

Thus, in this embodiment the database 21 stores sets of informationregarding the shape and/or appearance of the item 30 being assembled foreach stage in the production process. Information from the camera 16 onthe tool 1 being used to assemble the item is compared against thesesets of information by the computation unit 20. Although the computationunit 20 could attempt to compare the data captured by camera 16 againstthe entirety of the images in the database 21 (which collectivelyrepresent the appearance and/or shape of the object 30 at any point inthe process), this could involve a lengthy correlation/matchingcomputation which might be difficult to perform in real time.Furthermore, as the volume of data to be searched for a match increases,the probability of a false match also increases. In order to reduce thevolume of data to be searched, it is advantageous for the computationunit 20 to be guided in its search by information indicating the currentstage of the production process for the item 30. If this information isknown, the computation unit 20 can restrict its search for matches ofthe data from the camera 16 with only the potentially relevant sets ofinformation in the database 21.

There are a number of possible sources of information which can informthe computation unit 20 of the current stage of the production processfor the item 30.

-   1. If the tool 1 is a tethered tool, confined to a particular    station of the assembly line 37, then the computation unit 20 may be    configured to only attempt matches against information sets in the    database 21 that reflect the appearance and/or shape of the 30 item    at that point in the assembly process.-   2. The tool 1 may be located (at a lower level of accuracy than is    required for bolt-level location) by an auxiliary location system.    For example, the location system provided by units 24, 34, 35 and 36    may be used. That could be an ultrawideband radio system capable of    locating the tool 1 to an accuracy of around 15 cm in three    dimensions. That level of accuracy is sufficient to determine the    location in the assembly process of the particular item 30 under    assembly. This location is passed to the computation unit 20, which    uses it to determine the set of information in the database 21    against which data from the camera 16 should be compared. The    mapping from physical location to assembly process stage may be    stored in the computation unit 20 or the various image/shape    information sets in the database 21 may be indexed by location.-   3. Different workpieces travelling along the production line may be    adapted in different ways. For example, they may have different    engines or interior trims, meaning that they have different    appearances. The processor 20 may determine which appearance or    shape the workpiece is expected to have at a particular stage based    on a work schedule for the production line. The work schedule may    indicate which workpiece is at which stage of the production line,    and the build plan for that workpiece. The processor can then select    or prioritise images that relate to the workpiece scheduled to be on    the production line at the coarse location of the tool, and for the    state of build scheduled for the workpiece at that stage of the    production line. Alternatively, or in addition, the state of the    workpiece can be determined in dependence on work known to have been    carried out on it by tool 1 or other tools. As tools act on the    workpiece those actions can be stored, e.g. in the database 21. This    information can, for example, indicate that other tools have    previously performed certain bolt-tightening operations on the item    30, thus changing its appearance and/or shape. The processor 20 can    select or prioritise certain images for matching with a captured    image in dependence on that historic tool activity data.

This list is not exhaustive. It only illustrates examples of techniquesthat might be used.

On a production line there can be many different individual items inproduction at the same time. Any of the schemes described previously maybe extended so that the database 21 stores information about theshape/appearance and tool settings for each of the items currently onthe production line.

Other systems may inform the computation unit 20 of the currentlocations of each of the workpieces on a production line. These systemsmay include ultrawideband (UWB) tracking systems monitoring individualtracking tags (similar to tag 24) attached to the workpieces; orpositional encoders attached to the mechanism for moving the workpiecesalong the production line which, together with knowledge about when aparticular item was placed on the line, are sufficient to determine theworkpiece's location in the factory; or even manual processes. When aparticular tool 1 is used on a particular workpiece 30, the coarselocation of the tool 1 in the factory (determined, e.g., in one of theways described above) is compared by the computation unit 20 with theknown locations of workpieces 30 on the production line. From thatcomparison the computation unit 20 determines the individual workpiecewhich is located closest to the tool 1, and looks up in a datastore(e.g. 21 or 22) the current shape/appearance of that item in order toinform the process of fine-scale tool position determination usinginformation from the tool-borne camera 16. When a tool tighteningoperation has been performed on a bolt on the workpiece, the processor20 stores information from the tool 1 about the results of thetightening operation in the database 21 in relation to that individualworkpiece. In modern factories multiple different types of workpiece(e.g. different makes, models and colours of car) are produced on thesame production line. The appearance and shape of the individualworkpieces may differ. When a new workpiece 30 enters the productionline, the database 21 can be is updated with information about theshape/appearance (at each production stage) that should be expected ofthat particular item. Then, when a tool 1 is used on a particularworkpiece the computation unit 20 can use information about the currentlocations of the workpieces and the coarse or fine location of the tool1 to determine which individual workpiece the tool is operating upon.With that information images pertaining to the expected appearance/shapefor that individual workpiece at that stage of the production processcan be retrieved from the database 21 in order to assist the computationunit 20 to match an image from the camera 16.

Multiple tools may be working on a single workpiece and communicatingwith processor 20 to determine their position. An image received fromone tool may indicate that a part of the workpiece or its environment isobscured, for example by a hose, cable or a worker. The processor mayuse that information to assist future operations to determine thelocation of that or another tool. For example, it may estimate the zonewhere the obscuring object is located and subsequently prioritisematching features located elsewhere in the stored images. To achievethis, once the processor 20 has formed a match between a part of acaptured image and a part of a stored image it may then compare theremainder of the captured image with the remainder of the stored image.If it detects a substantial anomaly between those remainder parts of theimage it may take that anomaly to represent an obscuring object. It canthen store the location of the obscuring object and subsequently ignorematching parts of captured images to parts of stored images thatrepresent that location.

When a first tool is being operated on a workpiece, information from oneor more cameras attached to the tool are compared by a computation unit20 against a set of stored information regarding the appearance/shape ofthe item workpiece, in order to calculate the location of the head ofthe tool, so that the precise identity of a part (e.g. a bolt head) onwhich the tool is being used can be determined. There could be anobstruction (perhaps a cable draped across the workpiece) covering upsome of the workpiece. If there is a sufficient of the appearance/shapeof workpiece still visible to allow the location of the tool to bedetermined, the computation unit 20 may then determine (e.g. by aninability to match the captured scene in the obscured area of the fieldof view to the stored information in the database 21) that there arespecific areas of the image captured by the camera 16 which could relateto an obstruction. Since the computation unit 20 knows the location andorientation of the tool 1 and the camera 16, it can store informationregarding the location of the obscuring object. The computation unitcould even store in database 21 the image captured by the tool of theworkpiece or its environment with the obscuring object present. Then tolater estimate the location of the same or another tool the processorcan either ignore for matching purposes the area determined to beobscured, or it could attempt to match newly captured images to thestored image that includes the obscuring object. These techniques canpotentially increase the speed at which the computation unit 20 candetermine a match for the later-captured scene information and can alsoreduce the probability of a false match.

In a production environment such as a factory there may be many tools inoperation at one time. As described above, the local control unit 17 ofa tool transmits its video stream to a remote processing unit 20 foranalysis. When many tools are in operation, this can result insignificant amounts of data being transmitted, which can place strain onnetworks in the workplace. One approach to addressing that is to processlocally at a tool the image data captured by that tool. However, thiscan result in a need for significant processing power at the tool, whichmay be inefficient. Another approach will be described below. The toolis provided with a memory 40. Images captured by the tool's camera arestored in the memory 40. This may be done under the control of the localcontrol unit 17. The memory 40 may store a buffer of the most recentimages captured by the tool, and older images may be deleted from thememory. For example, the memory 40 may preferentially store imagescaptured during a time window of a predetermined length (e.g. 2 or 5seconds) before the present time. Those images may be stored as a videostream or as still images, or as more abstract data representing thecaptured image information. They may optionally be processed, e.g. bythe control unit 17, to simplify them by extracting or emphasisingsignificant features such as edges, contrast or predetermined colours.As new images are captured by the tool they are stored in the memory,and older images may be deleted. Normally, images are not transmittedfrom the tool to the remote processing unit 20. From time to time thetool is actuated. This may happen when the operator actuates operationswitch 14, or by the tool automatically starting the operation when itsenses that the toolpiece is correctly engaged with a workpiece.Alternatively, if the tool is a manually driven tool such as a manualwrench or screwdriver then actuation of the tool may be detected bydetecting a predetermined form of motion of the tool, e.g. rotation ofthe tool about an axis of symmetry of the toolpiece. In response to thetool being actuated, tool transmits to the remote processing unit 20data from the memory 40. This may be done under the control of the localcontrol unit 17. In one embodiment the local control unit may detectthat the tool has been actuated and respond to that determination bytransmitting the recently captured image data. In another embodiment thelocal control unit may detect that the tool has been actuated andrespond to that by transmitting to the remote processing unit 20 anindication that the tool has been actuated. The remote processing unitmay then request image data from the tool, and the tool may respond tothat request by transmitting to the remote processing unit the requestedimage data. In the latter approach, the remote processing unit mayinitially request the most recently captured image data from the tool,and subsequently—e.g. if the most recently captured image data is notsufficient for it to readily determine the tool's location—requestolder-captured image data from the tool. The information transmitted bythe tool comprises image data representing exclusively or preferentiallythe period of time immediately before the actuation of the tool. Forexample, in one embodiment the tool may transmit only image datarepresenting a predetermined period of time (e.g. 2 or 5 seconds) priorto the actuation of the tool. In another embodiment, the tool maytransmit at a first resolution image data representing a predeterminedperiod of time (e.g. 2 or 5 seconds) prior to the actuation of the tooland at a second, lower resolution image data representing time prior tothat predetermined period of time. The lower resolution image data maybe at a lower level of visual detail and/or at a lower capturefrequency. When the image data is received at the remote processing unitit estimates the position of the tool as described above. The positionof the tool may be estimated relative to the workplace or relative tothe workpiece on which the tool is working. Lower resolution image data(if present) may be used by the remote processing unit to help estimatethe general position of the tool. By selectively transmitting image datain response to actuation of the tool bandwidth for transmission of theimage data can be reduced.

In the examples described above, the tool has been described as a devicefor applying a torque to bolts. The tool could take other forms. Itcould for example be a screwdriver, nutrunner, drill, saw or spray gun.It could be manually operated: for example it could be a manual spanner,screwdriver or torque wrench. The part of the workpiece being acted onby the tool could take any suitable form. It could for example be abolt, screw, nut, stud, clip, hole or a bulk part of the workpiece to bedrilled, cut or coated.

The tool could have multiple cameras directed in different directions tohelp locate the tool irrespective of its orientation. It may be helpfulif a camera is directed towards the toolpiece since that may assist inlocating the toolpiece precisely, but that is not essential. The toolcould include a light for illuminating the workpiece and/or theenvironment around the workpiece to help the camera gather images towhich a match can readily be made.

The stored images could, as described above, be images captured ofexample workpieces. Alternatively, the stored images could be derivedfrom simulations, for example from three-dimensional CAD (computer-aideddesign) models of the workpiece and/or the environment. This may beparticularly helpful when the images are intended to represent aworkpiece that has many different production options. For example, a carmay have a large number of optional components and as a result there maybe a vast number of possible component combinations. Rather than captureimages of example cars in every possible combination, it may be betterto populate the database 21 with automatically generated simulatedimages for some or all component combinations. Three-dimensionalinformation for use with TOF cameras may be treated in the same way.

The images stored in database 21 may be direct photographicrepresentations. Alternatively, they may be simplified, for example byprocessing them to emphasise edge detail and/or de-emphasise texture.The reference images may be stored directly, as direct representationsdepicting the expected workpiece environment. This may, for example, bedone by storing digital images in an image format such as JPG, which canbe compared using image comparison software with images derived from thecaptured image from the tool. Alternatively, the reference images may bestored indirectly, as an indirect representation which does not depictthe workpiece environment visually but nevertheless represents it sothat a positional comparison can be made as described above. This may,for example, be done by forming a machine learning data set, e.g. bytraining a neural network to represent the expected workpieceenvironment, so that it can infer the location of the toolpiece, orwhether the toolpiece is correctly or wrongly positioned.

The tool may be actuated when the operator indicates that the requisiteoperation should proceed, e.g. by actuating switch 14, or by the toolautomatically starting the operation when it senses that the toolpieceis correctly engaged with the workpiece. At that point the correctsettings for the individual part of the workpiece to which the toolpieceis engaged can be applied, or have already been applied, to the tool,and the requisite operation is performed.

Information derived from the position of the tool may be used torestrict or inhibit the operation of the tool. For example, the motorcontroller 12 may be controlled by local controller 17 so as only toactivate the motor 13 when the toolpiece 6 is at a location where it isor can be correctly engaged with the workpiece. This can help avoiddamage to the workpiece, or injury to workers. The tool may be ahand-held tool. The tool may be an assembly tool.

The toolpiece may be non-removably attached to the tool. Alternatively,the tool may be capable of accepting multiple toolpieces: for examplehex sockets, screwdriver bits, drills, saws and paint canisters ofdifferent sizes and shapes. The toolpieces may attach to a mechanicalcoupling on the tool, for example on shaft 5. The mechanical couplingmay be a standard tool coupling, for example a ⅜″, ½″ or ¾″ square bossor a 5/16″ hex socket.

The link between the camera 16 and the processor 20 may be wireless, asillustrated in FIG. 1, or wired. The processor 20 could be located inthe tool, as could image database 21.

The processor 20 may be implemented as a single microprocessor ordistributed among multiple microprocessors at the same or differentlocations. The processor 20 can be configured to perform its functionsby executing software code stored non-transiently in a memory, forexample memory 22. Memory 22 and database 21 could be combined ordivided between any number of physical memory units at the same ordifferent locations.

In the process described above, the captured image is transformed to mapit to a stored image. Instead, the stored image could be transformed tomap it to the captured image, or both could be transformed. Thegeometric process of determining the position of the tool in dependenceon the transformation(s) that led to a match can then be adaptedaccordingly.

The applicant hereby discloses in isolation each individual featuredescribed herein and any combination of two or more such features, tothe extent that such features or combinations are capable of beingcarried out based on the present specification as a whole in the lightof the common general knowledge of a person skilled in the art,irrespective of whether such features or combinations of features solveany problems disclosed herein, and without limitation to the scope ofthe claims. The applicant indicates that aspects of the presentinvention may consist of any such individual feature or combination offeatures. In view of the foregoing description it will be evident to aperson skilled in the art that various modifications may be made withinthe scope of the invention.

1. A system configured to determine the location of a toolpiece, the system comprising: a tool carrying a toolpiece and comprising an imaging device configured to capture images of the environment around the tool, the tool being actuable to manipulate the toolpiece to work on a workpiece; and an image processor remote from the tool and communicatively coupled to the imaging device in order to receive images therefrom and having access to reference data representing an expected environment, the image processor being configured to compare an image captured by the imaging device with the reference data to identify a match therebetween and to determine in dependence on characteristics of that match the location of the toolpiece; wherein the system is configured to cause the tool to transmit images captured by the imaging device to the image processor in response to actuation of the tool and the tool is configured to inhibit the transmission of images captured by the imaging device to the image processor except in response to actuation of the tool.
 2. A system as claimed in claim 1, wherein: the image processor is configured to compare an image captured by the imaging device with the reference data by applying one or more geometric transformations to one or both of the image and the reference data so as to achieve a match therebetween; and the said characteristics of the match include the nature of the geometric transformations for which the match is achieved.
 3. A system as claimed in claim 1, wherein the reference data comprises reference images and the image processor has access to a reference location corresponding to each reference image, and the image processor is configured to determine the location of the toolpiece in dependence on the reference location for the at least one reference image matched to the image captured by the imaging device.
 4. A system as claimed in claim 3, comprising a data store storing the or each reference image and the or each corresponding reference location.
 5. A system as claimed in claim 3, wherein the reference images are photographs or simulation images.
 6. (canceled)
 7. A system as claimed in claim 1, wherein the reference data is a machine learning data set whereby an image captured by the tool can be associated with a respective reference location.
 8. A system as claimed in claim 1, wherein the imaging device is a camera capable of determining the range from the camera to a plurality of subjects represented in the image captured by the imaging device, and the reference data is a three-dimensional representation of an expected environment.
 9. A system as claimed in claim 1, comprising a memory storing a predetermined spatial relationship between the imaging device and the toolpiece, and wherein the processor is configured to determine the location of the toolpiece in dependence on the predetermined spatial relationship.
 10. A system as claimed in claim 1, wherein the tool has an automatically configurable operating mode, the system comprises a memory configured to store for each of a plurality of locations an indication of a respective operating mode, and the image processor is configured to determine whether the determined location of the toolpiece corresponds to one of the stored locations and if it does to configure the tool to operate in the operating mode indicated in respect of that location.
 11. A system as claimed in claim 10, wherein the operating mode is one of a maximum torque, a maximum force, an operating speed and an operating direction.
 12. A system as claimed in claim 10, wherein the image processor is configured to inhibit operation of the tool if the location of the toolpiece does not correspond to one of the stored locations.
 13. (canceled)
 14. A system as claimed in claim 1, comprising a secondary mechanism configured to estimate the location of the tool, and the image processor is configured to select a subset of the reference data for comparison with an image captured by the imaging device in dependence on a location of the tool estimated by the secondary mechanism.
 15. A system as claimed in claim 14, wherein the secondary mechanism comprises a radio transmitter comprised by the tool, a plurality of radio receivers configured to receive radio signals transmitted by the radio transmitter, and a processor configured to process the signals received by the radio receivers to estimate the location of the tool. 16-17. (canceled)
 18. A system as claimed in claim 1, wherein the image processor is configured to, on identifying a match between an image captured by the imaging device and a reference image, to compare non-matched portions of the captured image and the reference data, and on identifying non-matched portions that have corresponding offsets from matched portions of the respective images but that differ substantially from each other, determine those non-matched portions as representing an obscured part of the environment.
 19. A system as claimed in claim 18, wherein the image processor is configured to, on identifying an obscured part of the environment, subsequently ignore portions of the reference image(s) representing the obscured part of the environment when comparing an image captured by the imaging device with the reference data.
 20. A system as claimed in claim 18, wherein the image processor is configured to, on identifying an obscured part of the environment, store an image captured by the imaging device as reference data representing the obscured part of the environment.
 21. A system as claimed in claim 1, wherein the tool comprises at least one inertial motion sensor configured to sense motion of the tool, the tool is configured to provide output from the inertial motion sensor to the image processor, and the image processor is configured to, if it is inhibited from deriving a position for the toolhead in dependence on an image captured by the imaging device, derive a position for the toolhead in dependence on data from the motion sensor. 22-23. (canceled)
 24. A system as claimed in claim 1, wherein the system is configured to cause the tool to preferentially transmit images captured by the imaging device to the image processor in response to actuation of the tool.
 25. A system as claimed in claim 1, wherein the tool comprises a memory and the tool is configured to store in the memory images captured by the imaging device and to, in response to actuation of the tool, transmit to the image processor images stored in the memory.
 26. (canceled)
 27. A hand-held assembly tool comprising: a toolpiece or a mechanical coupling configured to attach to a toolpiece; a drive unit configured to drive the toolpiece or the coupling to move so as to perform an assembly operation; an imaging device configured to capture images of the environment around the tool; and a memory configured to store image data captured by the imaging device; and the tool is configured to transmit externally of the tool image data stored in the memory in response to actuation of the drive unit, wherein the tool is configured to inhibit the transmission of images captured by the imaging device except in response to actuation of the tool.
 28. (canceled) 